Texture mapping allows a colored, or textured, signal be imaged onto geometry, giving the appearance of detail on coarse triangle meshes. Using textures is generally more efficient than refining the carrier geometry to represent the signal per vertex. While rasterization hardware has supported basic texture mapping for many years, it has recently begun to offer more powerful features.
An overview of the texture mapping process is shown in FIG. 1. Given texture map 200, having texels 202, texture mapping applies the signal of the texture map 200 to triangles 210 of a mesh according to some geometric configuration of the triangle, and/or applies some additional transformation, such as shadowing, lighting, etc. which maps texels 202 to pixels 222.
Texture mapping is a problem that implicates the Nyquist theory, to the extent aliasing is to be avoided, which states that to accurately reproduce an analog waveform, the analog signal should be sampled at a rate at least twice its highest frequency to be able to represent the signal in the digital domain with a high level of accuracy. Digital-to-analog conversion techniques are then utilized to reconstruct the original analog signal accurately. If an insufficient sampling rate is used, the result is aliasing, which is incorrectly sampled high-frequency information that appears as lower frequency noise.
Sampling issues arise in connection with texture mapping as well. The process of texture mapping samples texture maps to screen pixels, and to the extent possible, the texture map data should have been sampled at a sufficiently high sampling rate to encompass the highest of frequency information. In this regard, typically, screen objects represented by a certain number of picture elements (pixels) use texture maps with a different number of texture elements (texels). When texture maps have more texels than the pixel space, a single pixel may map to more than one texel and when texture maps have fewer texels than the pixel space, multiple pixels may map to a single texel. When there is no exact mapping between texture space and picture space, display quality issues can arise.
In effect, the texture map may either not be able to effectively map proper image detail to the designated pixel area, or conversely, a smaller pixel area may not be able to use all the texture detail from a larger/richer texture map.
Specifically, in the case where a texture map is too small compared to the pixel area being mapped, then the same texel is mapped to adjacent pixels, which causes a blockiness effect when viewing the image. In the other case, if multiple texels map to the same pixel, it is often up for grabs which exact texel, from the group of texels that could map to a given pixel, will be selected from the texture map and applied to the pixel. It is thus an algorithm-dependent process and can result in artifacts known as texture swimming and pixel-popping, both of which are very noticeable when the camera moves. Moreover, even if a substantially similar or same number of pixels and texels exist for mapping therebetween, there may be some transformation, or function, applied to the texels prior to rendering, in which case some algorithm must still decide where the pixel data should come from with respect to the texel data.
For purposes of illustrating problem(s) that may arise, the case with a number of texels greater than the number of pixels may be considered as follows. Assume there a square billboard surface in a three-dimensional (3-D) scene comprising two equilateral triangles, wherein the square is 100×100 pixels. If one applied a texture map comprising 200 horizontal red and blue stripes on top of the 100×100 pixel square area, all 200 stripes of the texture could not be discerned once applied to the billboard because the texture has more samples than the pixel area to which it will be applied. Some of the intermediate texture data will not be used in the mapping, and will essentially be ignored.
As one's 3-D viewpoint is moved over the billboard, if navigating in a game, for example, a phenomena called pixel popping arises, which is a result of the texture's stripes coming in and out of alignment with the billboard pixels. This is a case of an insufficient texture-sampling rate to describe the scene's details. This is an illustration of one type of artifact that can results, as other types of artifacts can result from a poor texture sampling as well.
There are several approaches to minimize the rendering artifacts caused by an insufficient texture-sampling rate, mostly in the form of filtering and re-sampling. The two filter methods used in the simple texturing methods like point-sampling are Magnification and Minification, which alter the texture sampling methods to help avoid the problems depicted above.
The minification algorithm is used in the case where multiple texels can map to a single pixel, and it selects the best fit texel from among the group of texels that could map to the pixel. The magnification technique is used when multiple pixels can map to a single texel, and it maps a single texel to multiple pixels. There are many methods varying in speed and quality for applying textures, including point sampling, bilinear filtering, trilinear MIP-mapping, anisotropic filtering and antialiasing.
The most basic way to apply textures to a surface is point sampling, which uses a “nearest neighbor” sampling method. When the renderer fetches a piece of texture, it grabs the texture sample from the texture map that has (u, v) coordinates that map nearest to the pixel's coordinates (the pixel center), and applies it to the pixel. This approach is illustrated in FIG. 2A, wherein the relevant position of the mapping from pixel space to texel space is located at position P1. Here, since the closest texel is T1, T1 is selected. Though this approach requires the least amount of memory bandwidth in terms of the number of texels that need to be read from texture memory, i.e., one per pixel, the result often causes artifacts as discussed above, owing to insufficient samples, i.e., screen pixels, to describe the texture. But even first-generation personal computer (PC) 3-D hardware had a feature to help clean up, or at least hide these sampling problems to a certain extent, namely, by utilizing bilinear filtering.
Rather than just grab the nearest neighbor on a texture, bilinear filtering instead reads the four samples nearest to the pixel center, and uses a weighted average of those color values as the final texture color value. Thus, as illustrated in FIG. 2B, a weighted average of T1, T2, T3 and T4 is taken, wherein the weights are based on the distance from the pixel center to the four texel centers. Exemplary equations for bilinear filtering known to those of ordinary skill in the graphics processing art are as follows:ca=pij·α+(pij+1)·(1−αa)cb=pi+1j·α+(pi+1j+1)·(1−αa)F=lrp(ca, cb)·αb,where lrp is a linear interpolation function readily understood by today's graphics processing units (GPUs). The visual net effect is to blur out a good deal of the texture artifacts seen with point sampling, but because it is only a four-tap filter working with a single texture map, the effectiveness of bilinear filtering is limited.
Another approach to improving texture image quality is a technique known as MIP-mapping mapping, where the renderer makes multiple copies of the original texture, and each successive MIP-map is exactly half the resolution of the previous one. This effectively becomes a kind of 3-D texture, wherein the standard two coordinates, (u, v) are present, but there is also a third coordinate, d, used to measure which MIP-map(s) to select based on which map resolution will most closely match the pixel area to be mapped. In the case where a fighter plane is close to the view camera, a very detailed texture map would be selected, but as the plane flies off into the distance, successively smaller and less detailed maps would be used.
As the d coordinate increases, smaller and smaller MIP-maps are used. The derivation of the d coordinate tends to be complicated and implementation dependent, but in general relates to figuring out how much texture magnification or minification would result from a particular MIP-map selection based on the pixel area to be covered. The MIP-map with the smallest amount of texture magnification and minification is thus selected by this method. As illustrated by FIG. 2C, the problem of MIP-mapping is thus one of selecting which texture 200a, 200b, 200c, etc. to utilize to represent the number of pixels in pixel rendition 220 best.
There are a few different ways applications can use MIP-maps, and they are often dependent on graphics hardware support. In some cases, an application may opt (A) to select a single map that corresponds best to the pixel area to be mapped, (B) to apply bilinear filtering to texels in that map, and then (C) to apply the resulting averaged texels to their corresponding pixels. However, visual problems with bilinear MIP-mapping may occur at map boundaries when moving through the scene. For example, when driving down a roadway in a game, one can see visible breaks in roadway texture. This is likely due to a MIP-map switch.
A higher quality method of MIP-mapping often used is called trilinear MIP-mapping, which helps alleviate the MIP-map boundary problem discussed above. This filtering method takes two bilinear samples, using four texel samples each, from the two MIP-maps nearest to the pixel, where one texture map might be a bit larger in resolution and the other a bit smaller in resolution than the pixel area to be mapped. The trilinear algorithm then uses a weighted average to combine the two bilinear-filtered texels into a resultant textured pixel, with the map whose resolution is closer to the pixel area receiving more weight.
Even trilinear MIP-mapping suffers some shortcomings, however, mostly relating to level of detail (LOD) biasing. Each MIP-map is said to have an associated LOD, and biasing has to do with the weighting factor of the average between the two bilinear filtered samples taken from the two different maps. If LOD biasing is turned “up” too much, weighting the average more toward the smaller resolution map, texture blur can result. If the LOD bias is turned “down” too much, weighting the average toward the larger MIP-map, aliasing, and temporal aliasing, such as texture shimmering/crawling can result. LOD biasing thus alters the d value, causing smaller or larger maps to be used, although the d value is still preconfigured at development time. The main problem with these techniques is the isotropic nature of bilinear or trilinear filtering. The magnification/minification along one axis in the texture may be significantly different the magnification/minification along the other axis, which can result in aliasing along one axis and blurring in the other.
There is another filtering technique called anisotropic filtering that attempts to address the isotropic nature of bilinear and trilinear filtering. One form of anisotropic filtering uses MIP-map sampling, but rather than taking eight samples, it instead takes up to 16. In using anisotropic filtering, one specifies for the renderer what the degree of anisotropy is, which is a ratio of how far a texture can be stretched before its image quality starts to degrade.
In another form of anisotropic filtering, bilinear filtering is performed multiple times along some angle. This can be useful where “streaking” is a natural by product of the pixel rendering, e.g., as is the case with the road flying by in a race car video game. When the textured polygon is at an oblique angle to the plane of the screen, the footprint corresponding to a screen pixel mapped onto the texture becomes elongated, i.e., no longer square. More sophisticated methods use the odd-shaped collection of texels in this footprint and blend them together to produce the final result. Computing the correct shape to use for this sampling footprint, and how long it should be, is a very difficult problem. Any currently known algorithms for doing this base their computations only on the relative geometry of the screen-pixel to polygon texels alignment, and none provide flexibility that enables a developer to predictably/intelligently control the process.
This form of anisotropic sampling which multiply performs bilinear filtering is illustrated in FIG. 2D, for a point P1a, which is to be anisotropically filtered along an angle θ, defined by the ratio Δv/Δu, and a stepsize s between points P1a, P1b, P1c, etc., defined by the magnitude of Δv and Δu, in accordance with some prefixed geometric aspect of the triangle being rendered. An elliptical shape along the angle θ defines the positioning of P1b and P1c. Then, bilinear filtering is performed for each of P1a, P1b and P1c which for this example, a weighted fashion, blends together the corresponding groups of four: (A) T14, T11, T15 and T12, (B) T11, T12, T6 and T5 and (C) T7, T6, T8 and T1, respectively.
In using more samples in the direction the texture would have to be stretched to fit the surface where it is being applied, anisotropic filtering achieves sharper image quality than bilinear or trilinear filtering, and also avoids the texture shimmering found with point sampling. It is also noted that anisotropic-filtered text that is displayed at an angle, as might be in a billboard in a racing game, or at the start of a Star Wars movie, renders more clearly than with other filtering methods
Generally, graphics application programming interfaces (APIs) have provisions for a single vertex to store two or more texture addresses for use in multitexturing effects. Early on, multitexturing was used to create lighting effects using light maps, but it has since been applied to do additional effects like bump mapping. These multitexturing features eventually compelled 3-D chipmakers to invest chip real-estate into creating parallel pixel pipelines, and in more recent generations of chips, to process multiple texels per pixels per clock.
The advantage gained by having parallel pipes capable of handling multiple texels per pixel is that multitexturing operations can often be done in a single clock cycle, rather than having to perform multipass rendering, where each pixel in a scene is drawn several times. For each additional pass a chip has to make to render a scene, its effective pixel fill rate is divided by the number of passes that need to be made. So if a chip with 500Mpixels/sec of fill rate had to do two passes in order to render a scene with multitexturing effects in it, that chip's effective fill rate is halved to 250Mpixels/sec. If it makes four passes, the fill rate drops to 125Mpixels/sec. Thus, parallel pipelines have greatly increased the efficiency of graphics rendering as well.
Multitexturing does not operate that much differently from regular single texturing, except that the separate textures have to be blended together, and then blended with the existing pixel color produced by shading. However, doubling or tripling the number of textures being used in a scene drastically increases the amount of frame buffer bandwidth required to fetch the additional textures, and if multiple passes are being done, there once was concern that overall performance would suffer. However, the evolution of extremely fast graphics processors in recent years has laid these concerns to rest.
Texturing, and multitexturing, can be used to create a wide array of visual effects, with basic surface description being only the tip of the iceberg. Developers have also discovered another way to create textures; rather than store a “canned” bitmap, a developer can instead create a small program that creates textures procedurally. This technique has been used to create anything from marble surfaces, to terrain textures in large outdoor environments, to moving water effects.
Another process many applications seek to do with 3-D is to set a mood. This is especially important to attempt to immerse garners inside the world the developer has conjured up. One useful tool to achieve different moods is fog, which also helps give a scene an addition sense of depth of field.
Fog can be implemented in several different ways, with the variations having mostly to do with how the fog becomes foggier. Two very common methods are called linear and exponential fog, which scale linearly and exponentially, respectively, as their names suggest. Fog can also be applied per vertex or per pixel, and as in most things in 3-D, per pixel looks more convincing, but is computationally more expensive.
A more advanced method of applying fog is called range-based fog, which traditionally has not been available in consumer 3-D chips, but with the arrival of vertex shaders in DirectX8, range-based fog is an option. Another method is called fog table, where fog values are stored in a lookup table, then applied to each pixel.
But irrespective of technique, fog is a function of how far away an object is, which is usually determined by its z value, the distance from the view camera. A fog factor, whether computed linearly or exponentially, is then computed and applied to the pixel using a blending operation to combine the fog amount (color) and the lit/shaded/textured pixel color. If fog is being performed per vertex, fog operations become a part of the lighting calculation, and are interpolated across each triangle using Gouraud shading.
Another practical use for fog is to allow objects at the far clipping plane to gracefully “fade away” rather than just pop out of the scene. Fog can also be used to allow a renderer to only have to draw world objects relatively close to the view camera. So when the fog value gets near or reaches zero (fog values are typically floating-point values between zero and one, with zero being absolute fog and one being no fog), the object is essentially discarded, and no further rendering work needs to be done.
Techniques for applying transparency and translucency exist as well. Unlike opaque objects, transparent objects must generally be depth sorted in back-to-front order to ensure that the underlying colors that are blended with the transparent objects are available when the blending operation is performed. There are several formulae that get used to calculate an object's translucency, but a common formula is:co=α·cs+(1−α)·cd,wherein    co=final color of pixel    α=alpha value (between 0 and 1)    cs=color of the transparent pixel (called the source) and    cd=color of the occluded pixel (called the destination).
This formula is a sum of two pixel modulations, where the alpha value is applied differently to both pixels, and the resultant values are added together to give the final pixel color.
Thus, there are a myriad of techniques that can be applied in connection with texture sampling computer graphics; however, thus far, the selection of any of the myriad of techniques is fixed by the developer at development time without much flexibility. Since certain variables and heuristics are useful for certain graphical circumstances, and not others, the end quality result can be a mixed bag, even with ever increasing processing power. Accordingly, it would be desirable to increase the amount of intelligence applied to texture sampling algorithms at the front end of the process. In view of an increasing trend towards flex-function GPU, it would be desirable to expose controllable texture sampling for developers to apply on the fly, dynamic intelligence to handle different circumstances in the most efficacious manner. As more advanced programmable per-pixel shading and lighting algorithms become available that take into account material properties, curved surfaces, volumetric textures, and that have far more complex single-pass texture blending, it becomes all the more desirable to enable developers to have control over the texture sampling process. It might also be desirable to controllably introduce artifacts into a rendered image, such as blurring, or streaking, or shimmering. Accordingly, it would be desirable to provide any one or more of an operating system, driver code, an application programming interface, a tool kit and/or functionality in the graphics pipeline itself, including the GPU or other coprocessing device, a mechanism which provides controllable texture sampling capabilities.